Calibrating panoramic imaging system in multiple dimensions

ABSTRACT

Example apparatus and methods produce image correction data for a lens or sensor based on individual images acquired under different operating parameters. Example apparatus and methods then produce strip correction data based on a strip of images pieced together from the individual images. Example apparatus and methods then produce panoramic image correction data based on a panoramic image pieced together from two or more strips of images. The strip of images is produced without using a hemispherical mirror and thus accounts more accurately for issues associated with making a two dimensional representation of a three dimensional spherical volume. Images are acquired using different imaging parameters (e.g., focal length, pan position, tilt position) under different imaging conditions (e.g., temperature, humidity, atmospheric pressure, pan rate, tilt rate) to account for aberrations that may appear or be exacerbated under operating conditions. Example apparatus and methods then correct images using the correction data.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application62/062,310 filed Oct. 10, 2014.

BACKGROUND

A sub-optimally calibrated imaging system is inconvenient inrecreational applications. For example, a photograph from asub-optimally calibrated system may not faithfully or artisticallypresent the view a vacationer remembers. A sub-optimally calibratedimaging system may be fatal in military or security applications. Forexample, an image from a sub-optimally calibrated system that is used toactivate an alarm or weapons system may lead to catastrophicconsequences if ordinance or other measures are delivered inaccuratelyor inappropriately based on aberrations in the image. Thus, more optimalcalibration approaches that facilitate high-speed, high-resolution, widefield of view (FoV) imaging are constantly sought.

A panoramic imaging system may acquire multiple images (e.g., digitalphotographs) that when processed into a single image provide a largerfield of view than is available in a single image. For example, apanoramic imaging system may acquire multiple images that when processedtogether provide a three hundred and sixty degree view of an area aroundthe imaging system. Conventionally there have been different approachesfor acquiring the multiple images that are processed together into asingle image that has a larger field of view. One conventional approachto panoramic imaging includes acquiring images from several imageacquisition apparatus (e.g., cameras) that are pointed in differentdirections. Another conventional approach to panoramic imaging includesmoving a single image acquisition apparatus to different positions.Regardless of how the multiple images are acquired, the quality of thefinal image depends on several factors. One factor is the lens throughwhich the individual images are acquired.

Lenses are rarely, if ever, perfect. Lenses frequently have aberrationsthat reduce the fidelity of an image produced using the lens. Fidelity,as used herein, refers to the degree to which an electronic imagingsystem accurately reproduces a two dimensional image of a threedimensional scene from which the electronic imaging system receiveselectro-magnetic radiation as an input signal.

Calibrating an imaging system to account for an imperfection in a lensis well known for certain configurations. For example, calibrating animaging apparatus (e.g., camera) that will capture single images from asingle point of view (PoV) using a single field of view (FoV) producesinteresting problems whose solutions are well known. However,calibrating an imaging system that will capture multiple images frommultiple points of view using multiple fields of view while beingrelocated from position to position while operating parameters changeproduces more complicated problems for which conventional systemsprovide no answers.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate various example systems, methods,and other example embodiments of various aspects of the invention. Itwill be appreciated that the illustrated element boundaries (e.g.,boxes, groups of boxes, or other shapes) in the figures represent oneexample of the boundaries. One of ordinary skill in the art willappreciate that in some examples one element may be designed as multipleelements or that multiple elements may be designed as one element. Insome examples, an element shown as an internal component of anotherelement may be implemented as an external component and vice versa.Furthermore, elements may not be drawn to scale.

FIG. 1 illustrates an example panoramic imaging system.

FIG. 2 illustrates an example lens and sensor.

FIG. 3 illustrates an example lens and sensor.

FIG. 4 illustrates example images of a target item.

FIG. 5 illustrates an example lens and sensor.

FIG. 6 illustrates an example lens and sensor.

FIG. 7 illustrates an example lens and sensor.

FIG. 8 illustrates an example target pattern and actual lens projectionof the target pattern through a lens having aberrations.

FIG. 9 illustrates an example apparatus for calibrating a panoramicimaging system in multiple dimensions.

FIG. 10 illustrates an example apparatus for calibrating a panoramicimaging system in multiple dimensions.

FIG. 11 illustrates an example method associated with calibrating apanoramic imaging system in multiple dimensions.

FIG. 12 illustrates an example method associated with calibrating apanoramic imaging system in multiple dimensions.

DETAILED DESCRIPTION

Example apparatus and methods facilitate calibrating a panoramic imagingsystem in multiple dimensions under multiple operating conditions.Example apparatus and methods facilitate calibrating an imaging systemthat will capture multiple images from multiple points of view usingmultiple fields of view or multiple zoom levels under differentoperating conditions while being panned or tilted from position toposition at various rates or accelerations. Calibrating the panoramicimaging system facilitates accounting for aberrations in a lens orsensor used to acquire the images. Unlike conventional systems where alens and sensor may maintain a fixed relationship and may operate undersubstantially uniform operating conditions, an example system may changea relationship between the lens and sensor and may operate under a widerange of operating conditions. For example, the lens and sensor may behoused in a unit that weighs more than a hundred pounds, that isspinning at sixty revolutions per minute, that is being tilted whilespinning, that is simultaneously collecting electromagnetic radiation inmultiple spectra, and that is operating in temperature ranges from below0 F to over 100 F, in humidity ranges from substantially zero percent tosubstantially one hundred percent, in atmospheric pressure ranges fromless than one atmosphere to more than one atmosphere, and in varyinglight conditions. Calibrating the lens and sensor under this widevariety of operating conditions may reveal aberrations that are notdiscovered in single point of view static testing.

In one embodiment, calibration may be provided for a panoramic imagingsystem whose camera(s) pan to provide three hundred and sixty degrees ofhorizontal coverage and whose camera(s) tilt to provide at least onehundred and eighty degrees of vertical coverage. Thus, calibration maybe provided for a panoramic imaging system that acquires images thatcover a sphere centered at the location of the imaging system.Additionally, the zoom level or field of view for an image may vary asthe camera(s) are panned and tilted to different positions.Additionally, light conditions may be varied as the camera(s) arepanned, tilted, zoomed, and otherwise manipulated through the testspace.

Unlike conventional systems that may calibrate a camera for a fixed PoVor FoV for a single sensor type (e.g., photographic film, digitalimager) based on known or discovered lens properties, example apparatusand methods may calibrate multiple image acquisition assemblies (e.g.,lens plus sensor) for a dynamic PoV or FoV based on lens properties,sensor properties, and other properties under varying conditions. Unlikeconventional systems that may calibrate a camera for a single spectrum(e.g., visible light) at a single saturation, example apparatus andmethods may calibrate a system whose images may be combined with data orimagery from other sensors and for sensors that may experience a widevariety of light conditions. By way of illustration, an exampleapparatus may acquire images using visible spectra sensors, infraredspectra sensors, ultraviolet spectra sensors, laser range findingsensors, and other sensors. The camera(s) and other sensors may becombined into an apparatus that pans and tilts through ranges thatfacilitate covering an entire sphere or less than an entire sphere. Inone embodiment, the apparatus may rotate through three hundred and sixtydegrees acquiring a “line” of images at a first tilt angle and thenrotate through three hundred and sixty degrees again acquiring anotherline of images at a second tilt angle. The first two lines of images mayhave been acquired with a first zoom level that yielded a first FoV. Theapparatus may then rotate through one hundred and eighty degreesacquiring another line of images at a third tilt angle and using asecond zoom level that yielded a second FoV. Calibrating the imagingsystem to account for the horizontal movement, vertical movement,changing zoom levels or fields of view, and changing horizontal coverageproduces problems not encountered by conventional calibration systemsand reveals aberrations not discovered by conventional systems.

FIG. 1 illustrates an example panoramic imaging system 100. System 100may include an optical imaging assembly 110, a laser range findingapparatus 120, and a thermal imaging assembly 130. System 100 may bepanned using assembly 150 and may be tilted using assembly 140.Different panoramic imaging systems may include different numbers andtypes of sensors arranged in different configurations.

A conventional system that only has to consider light of a singleintensity in the visible spectrum for a single lens that will acquiresingle images at a single orientation and single zoom level may becalibrated from a single image. Conventional approaches include, forexample, displaying a grid on a surface that should produce a knownpattern in an acquired image, and then determining parameters for thelens that will correct for imperfections in the lens. Example apparatusthat have to consider inputs with multiple intensities from multiplespectra for acquisition apparatus that will acquire multiple images atmultiple orientations (pan, tilt), multiple depths of field (range),multiple fields of view, and multiple zoom levels, at different pan andtilt rates and rates of acceleration may not be calibrated so easily.The motion and acceleration of a unit may create or exacerbateaberrations that are not so apparent in a static system.

Conventional panoramic imaging systems may employ a linear array imagesensor that acquires just a thin narrow line of image data. This type ofconventional linear array image sensing does not produce warping effectsfor which calibration is necessary. Conventional panoramic imagingsystems may acquire image data for a scene having a single depth offield. This type of conventional image sensing does not producedifferent effects for different ranges to objects for which calibrationis necessary. Conventional panoramic imaging systems may acquire datausing a single zoom level. This type of conventional image sensing doesnot produce different effects for different zoom levels for whichcalibration is necessary. Conventional panoramic imaging systems mayacquire data using a single FoV. This type of conventional image sensingdoes not produce different effects for different fields of view forwhich calibration is necessary. Conventional panoramic imaging systemsmay acquire data using a stationary camera. This type of conventionalimaging sensing does not produce different blur effects for whichcalibration is necessary. Thus, example apparatus and methods facechallenges that do not exist for conventional stationary or evenpanoramic systems.

Conventional systems that produce single mode images (e.g., visiblelight only, IR only) do not face the same calibration issues thatsystems that produce multi-mode composite images (e.g., visible lightand IR). Different sensors may have different correction data that mayneed to be applied separately to individual sensors and then manipulatedfor the composite image.

In the field, at startup, an example system may automatically create aninitial, static panoramic image at the widest FoV settings available.This image may provide a user with a context that facilitates selectingworking parameters for the current deployment. The working parametersmay include, for example, a pan limit (e.g., azimuth range), a tiltlimit (e.g., elevation range), initial grid resolutions, range of gridresolutions, and other information. The setup parameters may define agrid of angular locations at which images will be acquired in the field.Example apparatus and methods may position the sensors to sample thefield at the angular locations. Samples acquired from the field may beused to select image correction parameters from the calibration data.

An image acquisition assembly may include a lens and a sensor. A lensmay have a focal length. Example image acquisition assemblies may beable to change the focal length for the assembly to zoom in or zoom out.Changing the zoom may also change the FoV. For example, when zoomed allthe way out, the FoV may have a first (e.g., larger) size and whenzoomed all the way in may have a second (e.g., smaller) size. The numberof pixels excited in a sensor may be the same regardless of the focallength, but the intensity of light acquired at a pixel may vary based onthe zoom level or focal length. The amount of space associated with apixel may also vary based on the zoom level or focal length. Forexample, the light from a three dimensional volume that is focused ontoa single sensor may come from volumes of different sizes. By way ofillustration, compare the light acquired from a volume when a camera islooking at a wide open dessert vista at high noon on June 21 in thenorthern hemisphere to the light acquired when the camera is looking atthe side of a building ten feet away at dawn on December 21 in thenorthern hemisphere. Conventional apparatus may not account for thisvolume effect. The light acquired at these times may also have adifferent angle. Conventional systems may not account for this angle.Example apparatus and methods may calibrate for varying zoom levels andfocal lengths which may produce volume effects for which calibration isperformed.

Calibration in the shop may produce data (e.g., mathematicalcoefficients) that characterizes the aberrations, if any, in a lens. Thecalibration may produce information about how a lens actually forms animage as opposed to how the theoretically perfect lens forms an image.Calibration in the shop may also produce data (e.g., mathematicalcoefficients) that characterizes the aberrations, if any, in a sensor.Calibration in the shop may also produce data (e.g., mathematicalcoefficients) for the combination of the lens and the sensor system.Calibration in the shop may be performed in isolation where the lens ischaracterized with a known sensor or where the sensor is characterizedwith a known lens. Calibration in the shop may also be performed in aproduction setting where the characterized lens and characterized sensorare mounted in the production unit to discover and remedy any effectsthat mounting the assembly in the production unit may produce. Forexample, a lens that receives light in the shop without a filter mayperform differently than a lens in the field that receives light througha cover. Like lenses may have aberrations, so too may covers, filters,or other optical components have aberrations. Thus, unlike conventionalsystems that may only calibrate in isolation in the shop, exampleapparatus and methods may calibrate in a multi-stage process that willproduce superior results.

Example apparatus may have image acquisition assemblies for which thefocal length can be changed. Acquiring information about the effectivefocal length facilitates subsequent image processing (e.g., stitchingtogether individual frames). Example apparatus and methods thereforeacquire calibration data that facilitates computing corrections to afield of view in an acquired image. The calibration data may be employedto determine de-warping corrections for an image. Different de-warpingcorrections may be needed for different images acquired when the imageacquisition assembly has different focal lengths.

A conventional lens and sensor assembly may have a single fixed focallength for which precise focal length determinations may be known.Example lens and sensor assemblies may have dynamic focal lengths due tozooming operations. Additionally, environmental factors (e.g.,temperature, humidity, air pressure) may affect focal length. Duringcalibration, position encoders may provide approximate focal lengthvalues, but small uncertainties may need to be anticipated andaddressed. For example, in the field, a sensor may experience dramatictemperature changes during a day. In the high desert, at midday,temperatures may exceed 120 F while at night temperatures may fall below32 F. Thus, significant thermal expansion or contraction may occur inthe lens, the sensor, and the apparatus in which the image acquisitionassembly is located. Example apparatus and methods may thereforecalibrate under varying temperature, pressure, humidity, or otherenvironmental conditions and a deployed unit may dynamically adapt itscorrection approach based on sensed conditions. Additionally, operatingparameters (e.g., pan rate, pan acceleration rate) may producemechanical forces that affect focal length or other properties. Forexample, a system that is spinning clockwise at 60 rpm may produce afirst mechanical stress that subtly changes the orientation of the lensand assembly in a first way while a system that is spinningcounterclockwise at 30 rpm may produce a second, different mechanicalstress that subtly changes the orientation of the lens and assembly in adifferent way. Example apparatus and methods may account for theseaberrations.

One calibration action involves determining optical distortion caused byaberrations in a lens. Different lens will have different aberrationsand thus different optical distortions. FIG. 2 illustrates a lens 200and an image sensor 210 arranged with different focal lengths 202 and204. There is an ideal theoretic path along which light arriving on path220 would arrive at image sensor 210. Similarly, there is an idealtheoretic path along which light arriving on path 230 would arrive atimage sensor 210. An actual lens may not exhibit ideal behavior. Thus,while light arriving along path 220 is illustrated arriving at sensor210 along the ideal theoretic path, light arriving on path 230 isillustrated arriving at sensor 210 along actual paths 231 and 232,neither of which are the ideal theoretic path. Conventional systems maycalibrate lens 200 and identify data (e.g., mathematical co-efficients)that can be applied to correct for the aberration that caused light totravel along actual paths 231 and 232 instead of ideal path 230. Themathematical co-efficients may be used in a transform that returns theactual lens to the performance of an ideal lens.

One calibration action involves determining optical distortion caused byaberrations in a sensor. Different sensors will have differentaberrations and thus different optical distortions. An aberration in asensor may result in a sensor reporting that light that should bereported as being at a point A is reported as being at a point a FIG. 3illustrates an example where a sensor 300 is reporting light 310 asbeing detected at point 314 instead of point 312 where the light 310 isactually being presented. The light may pass through lens 330 as thoughlens 330 was an ideal lens and yet sensor 300 may still incorrectlyreport the location of the detected light.

The aberration in the sensor 300, or an aberration in lens 330 mayresult from manufacturing, normal wear and tear, or actions that occurin the field. For example, a lens may get scratched or a sensor mayoverheat or freeze and malfunction under certain conditions.Conventionally, when calibration was performed once and only for a lens,post-manufacturing aberrations may not be addressed.

In some cases, damage may be so severe that recalibration cannot producecorrection values that will mitigate the effect of the aberration. Itmay be impractical to simply turn off the system and wait for areplacement unit. In these cases, the correction values may act toeliminate the data reported as a result of the aberration while allowingother data to be acquired. Rather than return light to where it issupposed to have been detected, correction values may remove the lightfrom the imaging process. This may facilitate keeping the panoramicimaging system in operation. Consider that a panoramic imaging systemstitches together multiple images to create the panoramic image. Even ifaberrations are too severe to correct, as long as a portion of an imagecan be acquired with adequate fidelity, multiple partial images maystill be stitched together to produce a useful panoramic image. Thisproduces the concrete, tangible, real-world result of improving theoperation of a panoramic imaging system under conditions that wouldcripple a conventional system.

Conventionally, the characterization of a lens and the calibration tofind data for transforming an actual lens back to an ideal lens isperformed using a fixed pattern synthetic target that is displayed on aflat surface at a single fixed, known distance and orientation from thelens being characterized. A typical synthetic target may be, forexample, a grid pattern having a high contrast between the pattern andthe background. An ideal lens will image straight lines from the targetplane as straight lines in an image plane, though the lines may not beparallel. A deviation from the straight lines identifies an aberrationin the lens. Mathematics may be performed to determine how to return thelines from their deviant position to a straight lines configuration.

FIG. 8 illustrates an example target pattern 800 and actual lensprojection 810 of the target pattern 800 through a lens havingaberrations. The straight lines in target pattern 800 have not beenfaithfully produced in the actual lens projection 810. The actual lensprojection 810 produces an image with less than perfect fidelity of thetarget pattern 800. Conventional systems may attempt to account for theresults of an aberration in a single dimension under a single set ofoperating parameters for a static system. Example systems are not solimited. Example apparatus and methods may account for the results ofaberrations that are revealed through testing in multiple dimensionsunder multiple sets of operating parameters for a dynamic system that isaccelerating and moving while acquiring images.

While conventional systems may acquire a single image in a singleorientation, example apparatus and methods are not so limited.Additionally, while conventional systems may acquire a single image of agrid on a flat surface, example systems and methods are not so limited.Conventional systems may anticipate that the lens will be used toproduce images that will be processed using a hemispherical mirrorapproach for single strip stitching. Example apparatus do not use thehemispherical mirror approach for single strip stitching. Instead, aspherical approach is employed.

Imagine you are standing inside a globe. Like the earth, the globe maynot be perfectly spherical in all dimensions. Consider the differentprocessing required to acquire a strip of images around the equator, toacquire a strip of images around the Tropic of Cancer, to acquire astrip of images around the Arctic Circle, and to acquire a strip ofimages very close to the North Pole. Acquiring a strip of images foreach of these meridians on the globe illustrates a three dimensionalprojection problem for which advanced calibration may be required. Alens and sensor assembly may experience a first set of aberrations whenacquiring images around a first meridian (e.g., equator) but mayexperience a second, different set of aberrations when acquiring imagesaround a second meridian (e.g., Arctic Circle). Even though the samelens and same sensor may be located in the same assembly, differentfocal lengths, different fields of view, different orientations in apositioning system, and other factors may contribute to differentaberrations for which different corrections are sought.

Now imagine that the globe in which you are standing has just expandedto twice its volume. The expansion may not have been uniform. Forexample the equator may have extended outwards away from the origin ofthe sphere while the poles may have actually moved closer together andcloser toward the origin of the sphere. Calibration data acquired forthe first globe may be sub-optimal for the second globe. Conventionalsystems may account for neither the three dimensional projection problemnor the changing sphere problem. Example apparatus and methods mayaccount for both.

Example apparatus and methods may acquire calibration images from a lensand sensor assembly that moves through six degrees of freedom includingtranslation in x, y, and z planes, as well as rotation about the axes toproduce pitch, roll, and yaw. In one embodiment, the calibration data isacquired “in the shop”. In another embodiment, calibration data may alsobe acquired “in the field.” “In the shop” refers to a situation wheremany parameters can be controlled using equipment or processes that areavailable for calibration. For example, “in the shop” may refer to in afacility (e.g., factory) where the lens or sensor are manufactured, in afacility where the subassembly is assembled, in a facility where thepanoramic imaging system is assembled, or other controlled orcontrollable environment. “In the field” refers to a situation where itmay be more difficult to control any parameters. Instead, calibrationperformed “in the field” may be performed with observed parametersinstead of controlled parameters.

Acquiring calibration data may include acquiring an image of a synthetictarget. The synthetic target may be spectrum specific. For example, acamera used to acquire light in the visible spectrum may acquire imagesof a visible grid of parallel and orthogonal lines. The lines may be,for example, black lines on a white background. Some images may beacquired from a grid on a flat surface while other images may beacquired from a grid on a portion of a spherical surface. A sensor usedto acquire signals from another spectrum (e.g., IR, NIR, SWIR, MWIR,LWIR) may acquire electromagnetic waves produced by an apparatus thatproduces parallel and orthogonal lines of IR radiation. Some images maybe acquired from a flat surface while other images may be acquired froma portion of a spherical surface. Lines in the images may then becurve-fit and the amount, if any, of curvature in a line may bedetermined. Information for correcting the curvature in a line may thenbe computed and stored. Images may be acquired while the lens and sensorassembly moves (e.g., translates, rotates) through the available degreesof freedom. In one embodiment, the synthetic target may also be moved(e.g., translated, rotated) through the available degrees of freedom.Images may be acquired under different conditions (e.g., temperature,humidity, atmospheric pressure, constant motion, non-constant motion).Images may also be acquired for different lens zoom positions thatproduce different focal lengths. The different lens zoom positions mayalso produce different FoV, although FoV may be manipulated in otherways. Thus, data that facilitates correcting the actual lens performanceto ideal or desired lens performance is acquired in a more robust andcomplicated manner than for conventional systems. The information isstored to facilitate correcting images on-the-fly when the system isoperating.

Calibrating a lens and sensor assembly may also include determining theeffective focal length of the lens. Improving information about theactual or effective focal length of the lens facilitates improvingstitching images together. In one embodiment, determining the effectivefocal length may include imaging a target of a known size at a knowndistance. When an item of known size is imaged at a known distance, thenthe actual focal length can be determined from the actual acquiredimage. Calibration data for the actual focal length may then be stored.In one embodiment, determining the actual focal length may includedetermining the numbers of pixels that are covered in differentdimensions by the target item when it is imaged at a known distance. Thefield of view, and thus the actual focal length, may then be determinedfrom the number of pixels that are covered. The actual or effectivefocal length may be determined for various configurations (e.g.,relative positionings) of the lens and sensor assembly. Exampleapparatus and methods may reposition the image acquisition assemblyaccording to the available degrees of freedom to make the calibrationmore thorough than conventional systems. Thus, rather than taking asingle image of a single item positioned directly in the center of theFob′ for the lens and sensor assembly, example apparatus and methods maymove the lens and sensor assembly so that the item is imaged indifferent locations in the FoV. For example, FIG. 4 illustrates an image400 that has representations of a target pattern at locations P1, P2,P3, P4, and P5. The image 400 may have been acquired by taking fiveimages with the target pattern 499 at different locations in the fieldof view of a camera. For example, images 402, 404, 406, and 408 could betaken so that target pattern 499 appears in the four different cornersof the resulting images.

FIG. 5 illustrates an object 540 being imaged by a lens 500 and sensor510 that are arranged to produce a focal length 502 for the assembly oflens 500 and sensor 510. Light 530 may pass through lens 500 and reachsensor 510 at one location while light 520 may pass through lens 500 andreach sensor 510 at another location. Conventional systems may take thissingle image to calibrate lens 500 or sensor 510. Example apparatus andmethods are not so limited.

FIG. 6 illustrates an object 640 being imaged in a different way thanjust the conventional approach illustrated in FIG. 5. FIG. 6 showsobject 640 positioned somewhere other than the center of the field ofview of lens 600 and sensor 610. Lens 600 and sensor 610 may have thesame focal length 602 as lens 500 and sensor 510 in FIG. 5. Light 620from object 640 may arrive at a different location on sensor 610 via adifferent path through lens 600 and light 630 from object 640 may arriveat a different location on sensor 610 via a different path through lens600. Imaging object 640 at different locations may facilitate exposingaberrations in different locations in lens 600. Conventional systems maytake calibration images at a single focal length or with objectspositioned at a consistent distance from the lens. Example apparatus andmethods are not so limited.

For example, FIG. 7 illustrates an object 740 positioned a greaterdistance from lens 700. Additionally, lens 700 and sensor 710 arearranged to produce a different focal length 702. Light 730 from object740 and light 720 from object 740 may arrive at sensor 710 via differentpaths through lens 700 than in conventional systems. Changing the depthat which object 740 is imaged and changing the focal length between lens700 and sensor 702 may facilitate revealing aberrations that are missedby conventional systems.

Determining the effective focal length may be performed in the shop orin the field. In one embodiment, an apparatus may include one or moretarget items that can be placed at known distances to facilitateperforming the effective focal length determination in the field.Calibration in the field may facilitate accounting for factorsincluding, for example, damage to a lens, damage to a sensor, heat,humidity, atmospheric pressure, wear and tear, or other factors. In oneembodiment, the effective focal length calibration may be performedwhile the system is in a staring mode (e.g., fixed pan angle, fixed tiltangle, fixed zoom). In another embodiment, the effective focal lengthcalibration may be performed while the system is in panoramic mode(e.g., changing pan angle, changing tilt angle, changing zoom, changingFoV).

During calibration, the lens and sensor assembly may be moved todifferent known pan angles and tilt angles and images of the target itemmay be taken at the different azimuths and elevations associated withthe pan angles and tilt angles. The angular information from positionencoders is used to determine the angular spatial extent that was imagedwhile the lens and sensor assembly were moved to the different azimuthsand elevations. The FoV may then be determined from the spatial extent,and the effective focal length may then be determined from the FoV.

During calibration, the lens and sensor assembly may be moved todifferent locations at different rates. Consider that a system acquiringimages using a calibrated lens and sensor assembly may be rotatingthrough three hundred sixty degrees every second. Consider also that thesystem may weigh one hundred pounds or more. While rotational motiontheoretically does not alter the properties of light or of a lens,accelerating, rotating, panning, or decelerating a machine that weighsover a hundred pounds may cause stresses on a lens, a sensor, alens/sensor assembly, or other components of an image acquisitionsystem. Therefore, example apparatus and methods may perform thecalibration while the system is static, while the system is acceleratingor decelerating at different rates, or while the system is moving at aconstant rate. Calibration data may be acquired and stored for thedifferent operating parameters.

FIG. 9 illustrates an apparatus 900 that facilitates calibrating apanoramic view imaging system. Apparatus 900 includes a processor 910, amemory 920, and a set 930 of logics that is connected to the processor910 and memory 920 by a computer hardware interface 940. In oneembodiment, processor 910 and the set of logics 930 may calibrate thepanoramic view imaging system under varying conditions that producesuperior results to conventional systems.

In one embodiment, the functionality associated with the set of logics930 may be performed, at least in part, by hardware logic componentsincluding, but not limited to, field-programmable gate arrays (FPGAs),application specific integrated circuits (ASICs), application specificstandard products (ASSPs), system on a chip systems (SOCs), or complexprogrammable logic devices (CPLDs). In one embodiment, individualmembers of the set of logics 930 are implemented as ASICs or SOCs. Inone embodiment, the first logic 931, the second logic 932, the thirdlogic 933, or the fourth logic 934 may be ASICs, FPGA, or otherintegrated circuits.

The set 930 of logics includes a first logic 931 that produces imagecorrection data for a lens associated with the panoramic imaging systemor a sensor associated with the lens and the panoramic imaging system.In one embodiment, the image correction data is based on an erroridentified in individual images acquired by the lens or sensor. Theindividual images are acquired with a plurality of pre-determinedoperating parameters. Different individual images may be acquired withdifferent values for members of the plurality of pre-determinedoperating parameters. The pre-determined operating parameters mayinclude, for example, horizontal position, vertical position, targetgrid co-ordinates, roll, pitch, yaw, field of view, focal length, depthof field, light intensity, angle of fight, volume of field, volume oflight, temperature, humidity, atmospheric pressure, pan rate, tilt rate,change in pan rate, or change in tilt rate.

In one embodiment, the individual image may be an image of a calibrationpattern. In different embodiments, the calibration pattern may be on aflat surface or may be on a portion of a spherical surface. In oneembodiment, to facilitate determining an effective focal length, theindividual image may be an image of a calibration item. The calibrationitem may have a known size and the image may be acquired while thecalibration item is positioned at a known distance from the lens.

The apparatus 900 also includes a second logic 932 that produces stripcorrection data for the lens or sensor. In one embodiment, the stripcorrection data is based on an error identified in a strip of imagespieced together from a plurality of individual images acquired by thelens or sensor. Certain anomalies associated with certain types ofaberrations may only become detectable or exceed a fidelity thresholdwhen images are stitched together. Since conventional systems may onlyproduce correction data for individual images, this type of anomaly oraberration may not be handled by conventional systems.

The apparatus 900 also includes a third logic 933 that producespanoramic image correction data for the lens or sensor. In oneembodiment, the panoramic image correction data is based on an erroridentified in a panoramic image pieced together from two or more stripsof images processed by the second logic 932. Certain anomaliesassociated with certain types of aberrations may only become detectableor exceed a fidelity threshold when strips of images are stitchedtogether. Since conventional systems may only produce correction datafor individual images, this type of anomaly or aberration may not behandled by conventional systems.

The apparatus 900 also includes a fourth logic 934 that storescorrection data in the memory 920. In different embodiments, the fourthlogic 934 may store the image correction data, the strip correctiondata, or the panoramic image correction data in the memory 920. In oneembodiment, the fourth logic 934 may store a combined correction valuethat is computed from the image correction data, the strip correctiondata, or the panoramic image data. The combined correction value mayfacilitate accounting for errors in individual images, in strips ofimages pieced together from individual images, or from panoramic imagesstitched together from strips of images. To facilitate retrievingappropriate correction data under different operating conditions, fourthlogic 934 also stores data that relates the pre-determined operatingparameters to the image correction data, the strip correction data, thepanoramic image correction data, or the combined correction value.

Apparatus 900 may perform the calibration for more than one sensor.Thus, in one embodiment, the first logic 931 produces image correctiondata for a second lens associated with the panoramic imaging system or asecond sensor associated with the lens and the panoramic imaging system.This image correction data may be based on an error identified in anindividual image acquired by the second lens or the second sensor. Likeit was for the first sensor, the individual image may be acquired with aplurality of pre-determined operating parameters. The first sensor maybe, for example, a sensor that operates on light in the visiblespectrum. The second sensor may be, for example, a sensor that operateson electromagnetic waves in other spectra (e.g., infrared (IR), near IR(NIR), short wave IR (SWIR), mid wave IR (MWIR), long wave IR (LWIR),ultraviolet). Different images may be acquired with differentpre-determined operating parameters.

In this embodiment, the second logic 932 also produces strip correctiondata for the second lens or the second sensor and the third logic 933produces panoramic image correction data for the second lens or thesecond sensor. With this additional correction data available for thesecond lens or sensor, the fourth logic 934 stores, in the memory 920,the correction data for the second lens or sensor. In one embodiment,combined correction data may be produced to facilitate correcting for acombined image that is produced from images from the two different typesof images.

FIG. 10 illustrates a panoramic view imaging system 1000 that can becalibrated in multiple dimensions. System 1000 includes a first imageacquisition assembly 1010 comprising a first lens 1012 and a firstsensor 1014 that produce a first image from light in a visible spectrum.In one embodiment, the first image acquisition assembly 1010 producesfirst images at a rate of at least sixty images per second. Images maybe acquired at other rates.

The system 1000 includes a rotational position controller 1020 that pansthe system 1000 or first image acquisition assembly 1010 through a rangeof horizontal imaging positions and an elevation position controller1030 that tilts the system 1000 or the first image acquisition 1010assembly through a range of vertical imaging positions. In oneembodiment, the range of horizontal imaging positions is three hundredand sixty degrees, and the range of vertical imaging positions is atleast one hundred and eighty degrees. Other ranges may be employed.

The apparatus 1000 also includes a zoom controller 1040 that changes thefocal length of the first image acquisition assembly 1010 by, forexample, moving the lens 1012 with respect to the sensor 1014 or viceversa. Zoom is just one parameter that may be manipulated duringcalibration. Other calibration parameters that can be manipulated mayinclude, for example, horizontal position, vertical position, targetgrid co-ordinates, roll, pitch, yaw, field of view, focal length, depthof field, volume of field, volume of light, light intensity, angle oflight, temperature, humidity, atmospheric pressure, pan rate, tilt rate,change in pan rate, or change in tilt rate.

The apparatus 1000 also includes an image processor 1050 that produces apanoramic image from a plurality of images produced by the first imageacquisition assembly 1010. The panoramic image has a field of viewgreater in both a horizontal dimension and a vertical dimension than asingle image acquired by the first image acquisition assembly 1010. Thepanoramic image is produced without using a hemispherical mirror.

The apparatus 1000 also has a calibration processor 1060 that calibratesthe panoramic view imaging system 1000. Calibrating the panoramic viewimaging system 1000 may involve several actions. The actions mayinclude, calibrating the first lens 1012 with a pre-characterizedsensor, calibrating the first sensor 1014 with a pre-characterized lens,calibrating the first lens 1012 and the first sensor 1014 arrangedtogether in the first image acquisition assembly 1010, or calibratingthe first image acquisition assembly 1010 mounted in the panoramic viewimaging system 1000. Conventional systems may only perform one of theseactions.

Calibrating the panoramic view imaging system 1000 facilitatesmitigating an effect of an aberration in the first lens 1012 or anaberration in the first sensor 1014. The aberrations may reduce thefidelity of an image produced by the panoramic view imaging system 1000.Calibrating the first lens 1012 with a pre-characterized sensor mayinclude acquiring a first plurality of images of at least one member ofa set of target patterns. The target patterns may include a grid patternon a flat surface, a grid pattern on a portion of a spherical surface,or other patterns. Members of the first plurality of images are acquiredusing the first lens 1012 and the pre-characterized sensor withdifferent calibration parameter values. For example, the lightintensity, light angle, position of the target image, orientation of thetarget image with respect to the lens 1012, or other parameters may bevaried. After acquiring the plurality of images, calibrating the firstlens 1012 may include identifying a first aberration in the first lens1012 from the first plurality of images. Once the aberrations areidentified, the calibrating may include determining a first set ofcorrection values for mitigating an effect of the first aberration inthe first lens 1012. The correction values may be used by a computerizedtransform process to change an acquired image to be closer to an idealimage. Once the correction values are computed they may be stored on acomputer-readable medium along with data that relates the first set ofcorrection values and the calibration parameter values.

Since an image may depend on both a lens and a sensor, both the lens1012 and the sensor 1014 may be calibrated separately with knowncomponents. In one embodiment, calibrating the first sensor 1014 withthe pre-characterized lens includes acquiring a second plurality ofimages of at least one member of the set of target patterns. Members ofthe second plurality of images are acquired using the pre-characterizedlens and the first sensor 1014 with different calibration parametervalues. A second aberration may be identified in the first sensor 1014from the second plurality of images. With the aberration identified, asecond set of correction values may be determined for mitigating aneffect of the second aberration in the first sensor 1014. In oneembodiment, the calibration may the proceed by selectively manipulatinga member of the first set of correction values stored on thecomputer-readable medium based, at least in part, on the second set ofcorrection values. In another embodiment the second set of correctionvalues may be stored separately on the computer-readable medium.

Some aberrations may produce anomalies when the lens 1012 and sensor1014 operate together. Thus, in one embodiment, the calibrating mayinclude calibrating the first lens 1012 and the first sensor 1014arranged together in the first image acquisition assembly 1010. Thiscalibrating may include acquiring a third plurality of images of atleast one member of the set of target patterns. The members of the thirdplurality of images are acquired using the first lens 1012 and the firstsensor 1014 arranged together in the first image acquisition assembly1010 using different values for the calibration parameters. For example,the focal length, the field of view, the pan position, tilt position, orother parameters may be varied. A third aberration associated with thefirst lens 1012 or a third aberration associated with the first sensor1014 may be identified from the third plurality of images and a thirdset of correction values for mitigating their effects may be produced.These correction values may then be used to manipulate a member of thefirst set of correction values stored on the computer-readable medium ormay be stored by themselves.

Forces may be produced by the movement and acceleration or decelerationof the panoramic view imaging system 1000. Thus, calibrating the firstimage acquisition assembly 1010 may include mounting the assembly 1010in the system 1000 and operating the system 1000 under differentoperating conditions. This calibrating may include acquiring a fourthplurality of images of at least one member of the set of targetpatterns. A fourth aberration in the first lens 1012 or a fourthaberration in the first sensor 1014 may be identified from the fourthplurality of images. A fourth set of correction values for mitigating aneffect of the fourth aberration in the first lens 1012 or the fourthaberration in the first sensor 1014 may then be computed and used toselectively manipulate a member of the first set of correction valuesstored on the computer-readable medium. In one embodiment, the fourthset of correction values may be stored by themselves.

While conventional systems may perform some calibration for a lens usinga single imaging approach, apparatus 1000 is not so limited. Thus, inone embodiment, the calibration processor 1060 may produce a strip ofimages from images acquired by the first image acquisition assembly1010. Producing the strip of images may include positioning the imagesrelative to each other based, at least in part, on pattern matching anitem visible in overlapping portions of the images. For example, an edgeor other distinguishable feature may be aligned in the overlap of thetwo images to position them correctly. Once they are positioned,additional anomalies associated with additional aberrations may bedetected. Thus, the calibrating may include identifying a fifthaberration in the first sensor 1014 or a fifth aberration in the firstlens 1012 from the fifth plurality of images. A fifth set of correctionvalues for mitigating an effect of the fifth aberration in the firstlens 1012 or the fifth aberration in the first sensor 1014 may then becomputed and used to selectively manipulate a member of the first set ofcorrection values stored on the computer-readable medium. In oneembodiment, the fifth set of correction values may be stored on thecomputer-readable medium.

Stitching images together into a strip may facilitate identifyingaberrations and anomalies that might not be otherwise detected.Stitching strips together into a panoramic image may also facilitateidentifying additional aberrations or anomalies. Thus, in oneembodiment, the calibration processor 1060 may produce a panoramic imagefrom strips of images. The panoramic image may be produced bypositioning the strips of images based, at least in part, on patternmatching of an item visible in the overlapping portions of the strips ofimages. For example, an edge or other feature may be used to align thestrips of images. Once the panoramic image has been produced, thecalibration may include identifying a sixth aberration in the firstsensor 1014 or a sixth aberration in the first lens 1012 from thepanoramic image. A sixth set of correction values for mitigating aneffect of the sixth aberration in the first lens 1012 or the sixthaberration in the first sensor 1014 may then be computed and used toselectively manipulate a member of the first set of correction valuesstored on the computer-readable medium. In one embodiment, the sixth setof correction values may be stored on the computer-readable medium.

While the arrangement of the lens 1012 and the sensor 1014 may have atheoretical focal length, the actual arrangement in a production systemoperating in the field may produce focal lengths that deviate from thetheoretical. Thus, in one embodiment, the calibration processor 1060 maycalibrate the panoramic view imaging system 1000 by acquiring, under aknown set of values for the calibration parameters, a test image of atarget item having a known size. The test image is acquired with thetarget item positioned at a known location (e.g., distance from lens1012) and thus values (e.g., dimensions, intensities) for the resultingimage may be anticipated. If the actual values differ from theanticipated values, then the actual focal length may be different thanthe theoretical focal length. In one embodiment, calibration may includedetermining an effective focal length of the first image acquisition1010 assembly based, at least in part, on a number of elements in thefirst sensor 1014 that receive light from the target item whileacquiring the test image. The effective focal length may then be used toselectively manipulate a member of the first set of correction valuesstored on the computer-readable medium. In one embodiment, the effectivefocal length may be stored on the computer-readable medium.

The calibration may be repeated for a second lens and a second sensor.The second lens and second sensor may operate in a different spectrumthan the first lens and the first sensor. In one embodiment, whencomposite images that combine imagery from multiple sensors areproduced, aberrations that only appear when the images are combined maybe detected and correction data for the composite imagery may beproduced. Once the correction values are acquired, they may be used tocorrect subsequently acquired images.

Thus, in one embodiment, the calibration processor 1060 may produce acombined image from the first panoramic image and the second panoramicimage and identify a combined aberration from the combined image. Thecalibration processor 1060 may then determine a combined correctionvalue to mitigate the effect of the combined aberration. The calibrationprocessor 1060 may then selectively manipulate correction values storedon the computer-readable medium based on the combined correction value.

After calibration, the panoramic view imaging system 1000 may be used toproduce panoramic view images of a scene. During operation, thecalibration processor 1060 acquires an initial image of a scene withknown calibration parameters, acquires a set of desired imagingparameters (e.g., zoom level, field of view), and acquires a set ofcurrent operating parameters (e.g., temperature, humidity, atmosphericpressure, pan rate, change in pan rate, tilt rate, change in tilt rate).The calibration processor 1060 may then retrieve a subset of correctionvalues from the computer-readable medium. The subset may be selectedbased, at least in part, on the set of desired imaging parameters andthe set of current operating parameters. For example, if the panoramicview imaging system is operating in a desert environment in brightsunlight and is scanning terrain up to several miles away, then a firstsubset of relevant correction values may be retrieved and stored in acache memory or register available to the image processor 1050.

The image processor 1050 will then acquire an individual image for usein producing the panoramic image and correct the individual image usinga member of the subset stored in the cache memory or register. By movingrelevant data closer to the image processor 1050, efficiency may beimproved. For example, fewer input/output operations may be required.Performing fewer input/output operations may improve speed whilereducing operating temperatures, which may in turn improve theefficiency of the system by requiring less cooling.

Conditions in the field may change, therefore, in one embodiment, upondetecting a change in the set of desired imaging parameters or the setof current operating parameters, the calibration processor 1060 mayacquire an updated set of desired imaging parameters, acquire an updatedset of current operating parameters, and selectively update the subsetin the cache memory or in the register based on the updated set ofdesired imaging parameters or the updated set of current operatingparameters. The updated set of desired imaging parameters may then beused to correct subsequently acquired images.

Some portions of the detailed descriptions herein are presented in termsof algorithms and symbolic representations of operations on data bitswithin a memory. These algorithmic descriptions and representations areused by those skilled in the art to convey the substance of their workto others. An algorithm, here and generally, is conceived to be asequence of operations that produce a result. The operations may includephysical manipulations of physical quantities. Usually, though notnecessarily, the physical quantities take the form of electrical ormagnetic signals capable of being stored, transferred, combined,compared, and otherwise manipulated. The physical manipulations create aconcrete, tangible, useful, real-world result.

It has proven convenient at times, principally for reasons of commonusage, to refer to these signals as bits, values, elements, symbols,characters, terms, or numbers. It should be borne in mind, however, thatthese and similar terms are to be associated with the appropriatephysical quantities and are merely convenient labels applied to thesequantities. Unless specifically stated otherwise, it is to beappreciated that throughout the description, terms including processing,computing, and determining refer to actions and processes of a computersystem, logic, processor, or similar electronic device that manipulatesand transforms data represented as physical (electronic) quantities.

Example methods may be better appreciated with reference to flowdiagrams. For purposes of simplicity of explanation, the illustratedmethodologies are shown and described as a series of blocks. However, itis to be appreciated that the methodologies are not limited by the orderof the blocks, as some blocks can occur in different orders orconcurrently with other blocks from that shown and described. Moreover,less than all the illustrated blocks may be required to implement anexample methodology. Blocks may be combined or separated into multiplecomponents. Furthermore, additional or alternative methodologies canemploy additional, not illustrated blocks.

FIG. 11 illustrates an example computerized method 1100 associated withcalibrating a panoramic view imaging system. Method 1100 can only beperformed in a computer because electronic voltages or other computersignals need to be generated to facilitate correcting electronic datavalues produced by a lens and an electronic sensor. These electronicvoltages or other computer signals cannot be generated by pen and paperor in the human mind. Method 1100 includes, at 1110, acquiring a firstframe from a panoramic view imaging system. The first frame is acquiredwith a pre-determined set of operating parameters. The operatingparameters may include, for example, horizontal position, verticalposition, target grid co-ordinates, roll, pitch, yaw, field of view,focal length, depth of field, light intensity, angle of light, volume offield, volume of light, temperature, humidity, atmospheric pressure, panrate, tilt rate, change in pan rate, or change in tilt rate.

Method 1100 also includes, at 1115, detecting an anomaly in the firstframe. The anomaly may be detected by, for example, examining thestraightness or other properties of lines in an image of a targetpattern. While patterns of horizontal intersecting lines are illustratedin FIG. 8, other types of patterns may be employed.

Method 1100 also includes, at 1120, determining first manipulation datathat facilitates correcting the first frame. The first manipulation datamay take the form of mathematical coefficients for which electronic datais stored. The electronic data may then be used to convert an acquiredimage to a corrected image that exceeds a fidelity threshold.

Method 1100 also includes, at 1125, acquiring a second frame from thepanoramic view imaging system. Unlike conventional systems, the secondframe is acquired with a second, different, pre-determined set ofoperating parameters than were used for the first frame. In oneembodiment, a calibration session may cycle through a variety ofpre-determined values to exercise the lens and sensor to revealaberrations or anomalies that may not be detected in conventionalsystems. Thus, actions 1110-1120 and 1125-1135 may be performedrepeatedly.

Method 1100 also includes, at 1130, identifying an anomaly in the secondframe. The anomaly may be detected by identifying a mismatch between anactual image and a desired image.

Method 1100 also includes, at 1135, determining second manipulation datathat facilitates correcting the second frame. The second manipulationdata may take the form of electronic signals that represent mathematicalvalues that may be used for a transform that is applied to an acquiredimage to make a corrected image that exhibits a desired level offidelity with a known calibration image or target pattern.

Method 1100 also includes, at 1140, stitching together a composite imagefrom the first frame and the second frame. The composite image has agreater field of view than either the first frame or the second frame.The composite image may be made by partially overlapping the first frameand the second frame. While a first frame and a second frame aredescribed, in one embodiment, a plurality of first frames or a pluralityof second frames may be acquired and stitched together to make thecomposite image.

Method 1100 also includes, at 1145, identifying an anomaly in thecomposite image. Certain anomalies that cause image fidelity to fallbelow a threshold may only be revealed when individual frames arestitched together into a composite image.

With the additional anomaly discovered, method 1108 then proceeds, at1150, by determining third manipulation data that facilitates correctingthe composite image.

Method 1100 also includes, at 1155, storing the first manipulation data,the second manipulation data, or the third manipulation data in acomputer-readable medium. The data may be stored in a fashion thatfacilitates retrieving different manipulation data that corresponds todifferent operating parameters. For example, records may be added to arelational database, entries may be made in a table, or values may bestored in linked lists or other data structures. These values may thenbe retrieved while the panoramic view imaging system is operating toproduce panoramic images in the field.

FIG. 12 illustrates another embodiment of method 1100. This embodimentuses the correction data produced in actions 1110 through 1150 tocorrect subsequently acquired images. Thus, this embodiment alsoincludes, at 1160, acquiring a third frame from the panoramic viewimaging system and, at 1165, correcting the third frame using the firstmanipulation data or the second manipulation data.

Method 1100 also includes, at 1170, acquiring a fourth frame from thepanoramic view imaging system and, at 1175, correcting the fourth frameusing the first manipulation data or the second manipulation data. Thusactions 1175 and 1165 concern correcting individual images usingcorrection data that was acquired during a calibration process. While asingle third frame and a single fourth frame are described, method 1100may include acquiring and correcting a plurality of images. Since theImages may be acquired under different operating conditions, differentcorrection data that corresponds to the operating conditions may beaccessed to correct the images. In one embodiment, the most relevantcorrection data may be stored in a computer memory that is fastest foran image processor to acquire. For example, a cache memory or registerin the image processor may be populated with relevant correction data.

Method 1100 also includes, at 1180, stitching together a secondcomposite image from the third frame and the fourth frame. The secondcomposite image has a greater field of view than either the third frameor the fourth frame. While a single third frame and fourth frame aredescribed, a plurality of images may be used to produce the secondcomposite image. The second composite image may then be corrected at1185 using the third manipulation data.

While the embodiment in FIG. 12 shows all of actions 1110 through 1185occurring in one sequence, in one embodiment, actions 1110 through 1155may be performed at a first time (e.g., during calibration in the shop)and actions 1160 through 1185 may be performed at a second time (e.g.,during operation in the field).

In one example, a method may be implemented as computer executableinstructions. Thus, in one example, a computer-readable medium may storecomputer executable instructions that if executed by a machine (e.g.,processor) cause the machine to perform method 1100. While executableinstructions associated with method 1100 are described as being storedon a computer-readable medium, it is to be appreciated that executableinstructions associated with other example methods described herein mayalso be stored on a computer-readable medium.

While example systems, methods, and other embodiments have beenillustrated by describing examples, and while the examples have beendescribed in considerable detail, it is not the intention of theapplicants to restrict or in any way limit the scope of the appendedclaims to such detail. It is, of course, not possible to describe everyconceivable combination of components or methodologies for purposes ofdescribing the systems, methods, and other embodiments described herein.Therefore, the invention is not limited to the specific details, therepresentative apparatus, and illustrative examples shown and described.Thus, this application is intended to embrace alterations,modifications, and variations that fall within the scope of the appendedclaims.

The following includes definitions of selected terms employed herein.The definitions include various examples and/or forms of components thatfall within the scope of a term and that may be used for implementation.The examples are not intended to be limiting. Both singular and pluralforms of terms may be within the definitions.

References to “one embodiment”, “an embodiment”, “one example”, “anexample”, and other similar terms, indicate that the embodiment(s) orexample(s) so described may include a particular feature, structure,characteristic, property, element, or limitation, but that not everyembodiment or example necessarily includes that particular feature,structure, characteristic, property, element or limitation. Furthermore,repeated use of the phrase “in one embodiment” does not necessarilyrefer to the same embodiment, though it may.

“Computer-readable storage medium”, as used herein, refers to anon-transitory medium that stores instructions and/or data. Acomputer-readable medium may take forms, including, but not limited to,non-volatile media, and volatile media. Non-volatile media may include,for example, optical disks, magnetic disks, and other disks. Volatilemedia may include, for example, semiconductor memories, dynamic memory,and other memories. Common forms of a computer-readable medium mayinclude, but are not limited to, a floppy disk, a flexible disk, a harddisk, a magnetic tape, other magnetic medium, an ASIC, a CD, otheroptical medium, a RAM, a ROM, a memory chip or card, a memory stick, andother media from which a computer, a processor or other electronicdevice can read.

“Data store”, as used herein, refers to a physical and/or logical entitythat can store data. A data store may be, for example, a database, atable, a file, a data structure (e.g. a list, a queue, a heap, a tree) amemory, a register, or other repository. In different examples, a datastore may reside in one logical and/or physical entity and/or may bedistributed between two or more logical and/or physical entities.

“Logic”, as used herein, refers to computer hardware or firmware, and/orcombinations of each to perform a function(s) or an action(s), and/or tocause a function or action from another logic, method, and/or system.Logic may include, for example, an instruction controlledmicroprocessor, a discrete logic (e.g., ASIC), an analog circuit, adigital circuit, a programmed logic device, or a memory devicecontaining instructions. Logic may include one or more gates,combinations of gates, or other circuit components. Where multiplelogical logics are described, it may be possible to incorporate themultiple logical logics into one physical logic. Similarly, where asingle logical logic is described, it may be possible to distribute thatsingle logical logic between multiple physical logics.

“Signal”, as used herein, includes but is not limited to, electricalsignals, optical signals, analog signals, digital signals, data,computer instructions, processor instructions, messages, a bit, or a bitstream, that can be received, transmitted and/or detected.

“User”, as used herein, includes but is not limited to one or morepersons, logics, applications, computers or other devices, orcombinations of these.

To the extent that the term “includes” or “including” is employed in thedetailed description or the claims, it is intended to be inclusive in amanner similar to the term “comprising” as that term is interpreted whenemployed as a transitional word in a claim.

To the extent that the term “or” is employed in the detailed descriptionor claims (e.g., A or B) it is intended to mean “A or B or both”. Whenthe applicants intend to indicate “only A or B but not both” then theterm “only A or B but not both” will be employed. Thus, use of the term“or” herein is the inclusive, and not the exclusive use. See, Bryan A.Garner, A Dictionary of Modern Legal Usage 624 (2d, Ed. 1995).

Although the subject matter has been described in language specific tostructural features or methodological acts, it is to be understood thatthe subject matter defined in the appended claims is not necessarilylimited to the specific features or acts described above. Rather, thespecific features and acts described above are disclosed as exampleforms of implementing the claims.

What is claimed is:
 1. A panoramic view imaging system, comprising: afirst image acquisition assembly comprising a first lens and a firstsensor that produce a first image from light in a visible spectrum,where the first image acquisition assembly produces first images at arate of at least sixty images per second; a rotational positioncontroller that pans the first image acquisition assembly through arange of horizontal imaging positions; an elevation position controllerthat tilts the first image acquisition assembly through a range ofvertical imaging positions; a zoom controller that changes the focallength of the first image acquisition assembly; an image processor thatproduces a panoramic image from a plurality of images produced by thefirst image acquisition assembly, where the panoramic image has a fieldof view greater in both a horizontal dimension and a vertical dimensionthan a single image acquired by the first image acquisition assembly,and where the panoramic image is produced without using a hemisphericalmirror; and a calibration processor that calibrates the panoramic viewimaging system by: calibrating the first lens with a pre-characterizedsensor; calibrating the first sensor with a pre-characterized lens;calibrating the first lens and the first sensor arranged together in thefirst image acquisition assembly; or calibrating the first imageacquisition assembly mounted in the panoramic view imaging system, wherecalibrating the panoramic view imaging system facilitates mitigating aneffect of an aberration in the first lens or an aberration in the firstsensor; a second image acquisition assembly comprising a second lens anda second sensor that produce a second image from electromagneticradiation in a spectrum outside the visible spectrum, where the secondimage acquisition assembly produces second images at a rate of at leastsixty images per second, and where the calibration processor calibratesthe panoramic view imaging system by: calibrating the second lens with asecond pre-characterized sensor; calibrating the second sensor with asecond pre-characterized lens; calibrating the second lens and thesecond sensor arranged together in the second image acquisitionassembly; or calibrating the second image acquisition assembly mountedin the panoramic view imaging system, where calibrating the panoramicview imaging system facilitates mitigating an effect of an aberration inthe second lens or in the second sensor; where calibrating the secondlens with the second pre-characterized sensor includes: acquiring afirst plurality of second images of at least one member of a second setof target patterns using the second lens and the secondpre-characterized sensor, where members of the first plurality of secondimages are acquired with a first set of calibration parameter values;identifying a first aberration in the second lens from the firstplurality of second images; determining a second set of first correctionvalues for mitigating an effect of the first aberration in the secondlens; and storing, on a computer-readable medium, the second set offirst correction values, the first set of calibration parameter values,and data that relates the second set of first correction values and thefirst set of calibration parameter values: where calibrating the secondsensor with the second pre-characterized lens includes: acquiring asecond plurality of second images of at least one member of the secondset of target patterns using the second pre-characterized lens and thesecond sensor, where members of the second plurality of second imagesare acquired with a second set of calibration parameter values;identifying a second aberration in the second sensor from the secondplurality of second images; determining a second set of secondcorrection values for mitigating an effect of the second aberration inthe second sensor; and updating a member of the second set of firstcorrection values stored on the computer-readable medium based, at leastin part, on the second set of second correction values; wherecalibrating the second lens and the second sensor arranged together inthe second image acquisition assembly includes: acquiring a thirdplurality of second images of at least one member of the second set oftarget patterns using the second lens and the second sensor arrangedtogether in the second image acquisition assembly, where members of thethird plurality of second images are acquired with a third set ofcalibration parameter values; identifying a third aberration in thesecond lens or a third aberration in the second sensor from the thirdplurality of second images; determining a third set of second correctionvalues for mitigating an effect of the third aberration in the secondlens or the third aberration in the second sensor; and updating a memberof the second set of first correction values stored on thecomputer-readable medium based, at least in part, on the third set ofsecond correction values; where calibrating the second image acquisitionassembly mounted in the panoramic view imaging system includes:acquiring a fourth plurality of second images of at least one member ofthe second set of target patterns using the second image acquisitionassembly mounted in the panoramic view imaging system, where members ofthe fourth plurality of second images are acquired with a fourth set ofcalibration parameter values; identifying a fourth aberration in thesecond lens or a fourth aberration in the second sensor from the fourthplurality of second images; determining a fourth set of secondcorrection values for mitigating an effect of the fourth aberration inthe second lens or the fourth aberration in the second sensor; andupdating a member of the second set of first correction values stored onthe computer-readable medium based, at least in part, on the fourth setof correction values: producing a second strip of images from two ormore second images acquired by the second image acquisition assembly,where producing the second strip of images includes positioning the twoor more second images based, at least in part, on pattern matching of anitem visible in the overlapping portions of the two or more images;identifying a fifth aberration in the second lens or a fifth aberrationin the second sensor from the second strip of images; determining afifth set of second correction values for mitigating an effect of thefifth aberration in the second lens or the fifth aberration in thesecond sensor; and updating a member of the second set of firstcorrection values stored on the computer-readable medium based, at leastin part, on the fifth set of second correction values; producing asecond panoramic image from two or more second strips of images, whereproducing the second panoramic image includes positioning the two ormore second strips of images based, at least in part, on patternmatching of an item visible in the overlapping portions of the two ormore second strips of images; identifying a sixth aberration in thesecond lens or a sixth aberration in the second sensor from the secondpanoramic image; determining a sixth set of second correction values formitigating an effect of the sixth aberration in the second lens or thesixth aberration in the second sensor; and updating a member of thesecond set of first correction values stored on the computer-readablemedium based, at least in part, on the sixth set of second correctionvalues: acquiring, under a known set of calibration parameter values, asecond test image of a target item having a known size, where the secondtest image is acquired with the target item positioned at a knownlocation; determining a second effective focal length of the secondimage acquisition assembly based, at least in part, on a number ofelements in the second sensor that receive electromagnetic radiation inthe spectrum outside the visible spectrum from the target item whileacquiring the second test image, and updating a member of the second setof first correction values stored on the computer-readable medium basedon the second effective focal length.
 2. The panoramic view imagingsystem of claim 1, where the range of horizontal imaging positions isthree hundred and sixty degrees, and where the range of vertical imagingpositions is at least one hundred and eighty degrees.
 3. The panoramicview imaging system of claim 1, where calibrating the first lens withthe pre-characterized sensor includes: acquiring a first plurality ofimages of at least one member of a set of target patterns, where membersof the first plurality of images are acquired using the first lens witha first set of calibration parameter values and the pre-characterizedsensor with a second set of calibration parameter values; identifying afirst aberration in the first lens from the first plurality of images;determining a first set of correction values for mitigating an effect ofthe first aberration in the first lens; and storing, on acomputer-readable medium, the first set of correction values, the firstand second sets of calibration parameter values, and data that relatesthe first set of correction values and the first and second sets ofcalibration parameter values.
 4. The panoramic view imaging system ofclaim 3, where the first set of calibration parameters includehorizontal position, vertical position, target grid co-ordinates, roll,pitch, yaw, field of view, focal length, depth of field, volume offield, volume of light, light intensity, angle of light, temperature,humidity, atmospheric pressure, pan rate, tilt rate, change in pan rate,or change in tilt rate.
 5. The panoramic view imaging system of claim 3,where the set of target patterns include a grid pattern on a flatsurface or a grid pattern on a portion of a spherical surface.
 6. Thepanoramic view imaging system of claim 4, where calibrating the firstsensor with the pre-characterized lens includes: acquiring a secondplurality of images of at least one member of the set of targetpatterns, where members of the second plurality of images are acquiredusing the pre-characterized lens with a third set of calibrationparameter values and the first sensor with a fourth set of calibrationparameter values; identifying a second aberration in the first sensorfrom the second plurality of images; determining a second set ofcorrection values for mitigating an effect of the second aberration inthe first sensor; and updating a member of the first set of correctionvalues stored on the computer-readable medium based, at least in part,on the second set of correction values.
 7. The panoramic view imagingsystem of claim 6, where calibrating the first lens and the first sensorarranged together in the first image acquisition assembly includes:acquiring a third plurality of images of at least one member of the setof target patterns, where members of the third plurality of images areacquired using the first lens and the first sensor arranged together inthe first image acquisition assembly with a fifth set of calibrationparameter values; identifying a third aberration in the first lens or athird aberration in the first sensor from the third plurality of images;determining a third set of correction values for mitigating an effect ofthe third aberration in the first lens or the third aberration in thefirst sensor; and updating a member of the first set of correctionvalues stored on the computer-readable medium based, at least in part,on the third set of correction values.
 8. The panoramic view imagingsystem of claim 7, where calibrating the first image acquisitionassembly mounted in the panoramic view imaging system includes:acquiring a fourth plurality of images of at least one member of the setof target patterns, where members of the fourth plurality of images areacquired using the first image acquisition assembly mounted in thepanoramic view imaging system with a sixth set of calibration parametervalues; identifying a fourth aberration in the first lens or a fourthaberration in the first sensor from the fourth plurality of images;determining a fourth set of correction values for mitigating an effectof the fourth aberration in the first lens or the fourth aberration inthe first sensor; and updating a member of the first set of correctionvalues stored on the computer-readable medium based, at least in part,on the fourth set of correction values.
 9. The panoramic view imagingsystem of claim 8, where the calibration processor: produces a strip ofimages from two or more images acquired by the first image acquisitionassembly, where producing the strip of images includes positioning thetwo or more images based, at least in part, on pattern matching an itemvisible in overlapping portions of the two or more images; identifies afifth aberration in the first sensor or a fifth aberration in the firstlens from a fifth plurality of images; determines a fifth set ofcorrection values for mitigating an effect of the fifth aberration inthe first lens or the fifth aberration in the first sensor; and updatesa member of the first set of correction values stored on thecomputer-readable medium based, at least in part, on the fifth set ofcorrection values.
 10. The panoramic view imaging system of claim 9,where the calibration processor: produces a panoramic image from two ormore strips of images, where producing the panoramic image includespositioning the two or more strips of images based, at least in part, onpattern matching of an item visible in the overlapping portions of thetwo or more strips of images; identifies a sixth aberration in the firstsensor or a sixth aberration in the first lens from the panoramic image;determines a sixth set of correction values for mitigating an effect ofthe sixth aberration in the first lens or the sixth aberration in thefirst sensor; and updates a member of the first set of correction valuesstored on the computer-readable medium based, at least in part, on thesixth set of correction values.
 11. The panoramic view imaging system ofclaim 1, where the calibration processor calibrates the panoramic viewimaging system by: acquiring, under a known set of calibration parametervalues, a test image of a target item having a known size, where thetest image is acquired with the target item positioned at a knownlocation; determining an effective focal length of the first imageacquisition assembly based, at least in part, on a number of elements inthe first sensor that receive light from the target item while acquiringthe test image, and updating a member of the first set of correctionvalues stored on the computer-readable medium based on the effectivefocal length.
 12. The panoramic view imaging system of claim 1, wherethe calibration processor: produces a combined image from the firstpanoramic image and the second panoramic image; identifies a combinedaberration from the combined image; determines a combined correctionvalue to mitigate the effect of the combined aberration; and updates amember of the first set of correction values based on the combinedcorrection value, or updates a member of the second set of firstcorrection values based on the combined correction value.
 13. Thepanoramic view imaging system of claim 11, where the calibrationprocessor: acquires an initial image of a scene, where the initial imageis acquired with known calibration parameters; acquires a set of desiredimaging parameters; acquires a set of current operating parameters;retrieves a subset of the first set of correction values from thecomputer-readable medium, where the subset is selected based, at leastin part, on the set of desired imaging parameters and the set of currentoperating parameters; and stores the subset in a cache memory orregister available to the imaging processor, and where the imageprocessor: acquires an individual image for use in producing thepanoramic image; and corrects the individual image using a member of thesubset stored in the cache memory or register.
 14. The panoramic viewimaging system of claim 13, where the calibration processor, upondetecting a change in the set of desired imaging parameters or the setof current operating parameters: acquires an updated set of desiredimaging parameters; acquires an updated set of current operatingparameters; and selectively updates the subset in the cache memory or inthe register based on the updated set of desired imaging parameters orthe updated set of current operating parameters.
 15. An apparatus forcalibrating a panoramic imaging system, comprising: a processor; amemory; a set of logics; and a hardware interface that connects theprocessor, the memory, and the set of logics; the set of logicscomprising: a first logic that produces image correction data for a lensassociated with the panoramic imaging system or a sensor associated withthe lens and the panoramic imaging system, where the image correctiondata is based on an error identified in an individual image acquired bythe lens or sensor, where the individual image was acquired with aplurality of pre-determined operating parameters; a second logic thatproduces strip correction data for the lens or sensor, where the stripcorrection data is based on an error identified in a strip of imagespieced together from a plurality of individual images acquired by thelens or sensor; a third logic that produces panoramic image correctiondata for the lens or sensor, where the panoramic image correction datais based on an error identified in a panoramic image pieced togetherfrom two or more strips of images processed by the second logic; and afourth logic that stores, in the memory, the image correction data, thestrip correction data, the panoramic image correction data, or acombined correction value computed from the image correction data, thestrip correction data, and the panoramic image data, and data thatrelates the pre-determined operating parameters to the image correctiondata, the strip correction data, the panoramic image correction data, orthe combined correction value, where the pre-determined operatingparameters include horizontal position, vertical position, target gridco-ordinates, roll, pitch, yaw, field of view, focal length, depth offield, light intensity, angle of light, volume of field, volume oflight, temperature, humidity, atmospheric pressure, pan rate, tilt rate,change in pan rate, or change in tilt rate.
 16. The apparatus of claim15, where the individual image is an image of a calibration pattern,where the calibration pattern is on a flat surface or on a portion of aspherical surface; or where the individual image is an image of acalibration item, where the calibration item has a known size, and wherethe individual image is acquired while the calibration item ispositioned at a known distance from the lens.
 17. The apparatus of claim16, where the first logic produces second image correction data for asecond lens associated with the panoramic imaging system or a secondsensor associated with the lens and the panoramic imaging system, wherethe second image correction data is based on an error identified in anindividual image acquired by the second lens or the second sensor, wherethe individual image was acquired with a plurality of pre-determinedoperating parameters; where the second logic produces second stripcorrection data for the second lens or the second sensor, where thesecond strip correction data is based on an error identified in a stripof images pieced together from a plurality of individual images acquiredby the second lens or the second sensor; where the third logic producessecond panoramic image correction data for the second lens or the secondsensor, where the second panoramic image correction data is based on anerror identified in a panoramic image pieced together from two or morestrips of images processed by the second logic; and where the fourthlogic stores, in the memory, the second image correction data, thesecond strip correction data, the second panoramic image correctiondata, or a combined second correction value computed from the secondimage correction data, the second strip correction data, and the secondpanoramic image data, and data that relates the pre-determined operatingparameters to the second image correction data, the second stripcorrection data, the second panoramic image correction data, or thecombined second correction value.